Automatic batching of gui-based tasks

ABSTRACT

Described herein are techniques for automatically batching GUI-based (Graphical User Interface) tasks. The described techniques include automatically determining whether a user is performing batchable tasks in a GUI-based environment. Once detected, the described techniques include predicting the next tasks of a batch based upon those detected batchable tasks. With the described techniques, the user may be asked to verify and/or correct the predicted next tasks. Furthermore, the described techniques may include performing a batch and doing so without user interaction.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 14/579,932, filed on Dec. 22, 2014,entitled, “Automatic Batching of GUI-Based Tasks,” which is acontinuation of, and claims priority to, Ser. No. 13/756,237, filed onJan. 31, 2013, which is a continuation of, and claims priority to, U.S.patent application Ser. No. 12/948,353, filed on Nov. 17, 2010, thedisclosures of which are incorporated herein by reference in theirentirety.

BACKGROUND

During their daily usage of a computer, users often perform manyrepetitive tasks that are batchable. An example of batchable tasksincludes repeating essentially the same re-size operation on all of thephotos in a particular folder on a storage device. A “batchable task” isan operation (e.g., re-size) that is repeated, but where a few aspects(e.g., specific photo file in a folder) vary with each operation.Batchable tasks may include essentially the same action repeated or mayinvolve a repeated pattern of differing actions.

Some surveys have suggested that employees may spend as much as an hourin a typical workday performing repetitive operation, such as batchabletasks. Of course, a computer can be, and often is, programmed to performbatchable tasks. Unfortunately, many users do not possess sufficienttechnical savvy to program such tasks. Even if a user has the technicalsavvy to script a series of batchable tasks, many of those savvy usersdo not bother to take the time to write an appropriate script to performa batch of the batchable tasks.

Moreover, few, if any, options exists for scripting a series ofbatchable tasks across multiple applications available on one of themany available GUI-based (Graphical User Interface) Operating System(OS) environments. Present solutions offer application-specific “macro”scripting capabilities within particular applications. For example, anapplication-specific macro can be run within a word processingapplication, such as Microsoft® Word. However, as the label suggests,application-specific macros do not operate across multiple applicationsin a GUI-based OS environment.

Furthermore, no present solution offers a way of automaticallydetermining that a user is performing batchable tasks in a GUI-based OSenvironment, and then automatically completing a batch started by theuser.

SUMMARY

Described herein are techniques for automatically batching GUI-based(Graphical User Interface) tasks. The described techniques includeautomatically determining whether a user is performing batchable tasksin a GUI-based environment. Once detected, the described techniquesinclude predicting the next tasks of a batch based upon those detectedbatchable tasks. With the described techniques, the user may be asked toverify and/or correct the predicted next tasks. Furthermore, thedescribed techniques may include performing a batch, and doing sowithout user interaction.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The term “techniques,” for instance, may refer to device(s),system(s), method(s) and/or computer-readable instructions, as permittedby the context above and throughout this document.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Thesame numbers are used throughout the drawings to reference like featuresand components.

FIG. 1 illustrates an example computing environment that implementstechniques to facilitate automatic batching of batchable tasks in aGUI-based (Graphical User Interface) Operating System (OS) environment.

FIG. 2 illustrates in more detail a module configured to implement thetechniques described herein to facilitate the automatic batching ofbatchable GUI-based tasks.

FIGS. 3 and 4 are flow diagrams of one or more example processes, eachof which implements the techniques described herein.

DETAILED DESCRIPTION

Described herein are techniques for automatically batching GUI-based(Graphical User Interface) tasks. The described techniques includeautomatically determining whether a user is performing batchable tasksin a GUI-based environment. Once detected, the described techniquesinclude predicting the next tasks of a batch based upon those detectedbatchable tasks. In other words, after a pattern of batchable tasks isidentified, a system using the described techniques may project what oneor more of the next few tasks might be.

The user may be asked to verify and/or correct the predicted next tasks.After verification or correction, the described techniques may includeautomatically (i.e., without user interaction) performing a batch basedat least in part upon the corrected or verified next tasks.

The following is an example image “re-sizing” scenario that mightillustrate how one or more described techniques may be implemented. Inthis example scenario, a user's computer system has a particularsubfolder containing one hundred digital photos. Using the GUI-based OS,the user navigates to view the contents of the particular subfolder.Using a GUI-based image-manipulation application, the user selects thefirst digital photo in the subfolder and re-sizes the image to meet theuser's specific criteria (e.g., perhaps to be better viewed on a smallerdisplay device). With the conventional approach, the user must repeatthe re-sizing task one hundred times (i.e., once for each digitalphoto). Each of the iterations of the re-sizing task is essentially thesame as each of the other iterations. The only differences betweeniterations are a few details, such as the name of the particular digitalphoto being re-sized in that iteration.

If the user is sufficiently trained, the user could write a short scriptor generate an application-specific macro to perform the re-sizing taskone hundred times to process the one hundred digital photos in thesubfolder. Most users do not have the skills to write such a script, andthose that do have the skills may not want to take the time to writesuch a script because it might take nearly as long to write as it wouldto manually perform the repetitive tasks.

With at least one described implementation, a configured system tracksthe GUI-based re-sizing tasks as the user manually performs each task.Once it identifies a pattern of tracked tasks, the system determinesthat the tasks of the pattern are capable of being batched. Such tasksare called “batchable tasks” herein. Since the user appears to beselecting digital photos in a particular subfolder, it is likely thatthe user intends to re-size each of the remaining digital photos in thatsubfolder. The system may predict the next few tasks based upon therepeated pattern and anticipating which of the digital photos are nextin the subfolder.

By visually walking the user through the next one or more predictedtasks, the user is given a chance to verify whether the predicted nextfew tasks are correct. Once verified, the system generates the fullbatch (based upon the verified prediction) and the batch runsautomatically—without manual user input—thereafter. In this examplescenario, the remaining digital photos in the subfolder are re-sizedwithout any additional manual input from the user.

Example Computing Environment

FIG. 1 illustrates an example computing environment 100 that mayimplement the described techniques for automatically batching GUI-basedtasks. The environment 100 may include at least one computing device102, which may be coupled together via a network 104 to form adistributed system with other devices. A user 106 may operate thecomputer device 102 to conduct work, to search for information, to writean email, to edit a video, to create a business presentation, or othersuch actions. Although not necessarily illustrated, the computing device102 has input/output subsystems, such as a keyboard, mouse, speakers,etc. The network 104, meanwhile, represents any one or combination ofmultiple different types of networks interconnected with each other andpossibly functioning as a single large network (e.g., the Internet or anintranet). The network 104 may include wire-based networks (e.g., cable)and wireless networks (e.g., cellular, satellite, etc.).

The computing device 102 of this example computer environment 100includes one or more processors 108, one or more storage subsystems 110,at least one monitor 112, and one or more memories 114. Running, atleast in part, on the computing device 102 is an Operating System (OS)116 and, in particular, this OS interacts with the user 106 via aGraphical User Interface (GUI). Herein, this OS is called the GUI-basedOS 116. An example of a typical GUI-based user interface (UI) 118 of aGUI-based OS 116 is shown on the monitor 112. This example GUI-based UI118 includes two example GUI-based applications 120 and 122 that mightbe displayed on such a UI.

FIG. 1 shows a module 124 for automatically batching GUI-based tasks,which is called the auto-batcher 124 herein. This module is shown asresiding in the memory 114 as a module of computer-executableinstructions. But, of course, in alternative embodiments, theauto-batcher 124 may be implemented in hardware, firmware, or somecombination of hardware, software, or firmware. The auto-batcher 124 maybe implemented as part of the GUI-based OS 116 or running under or incooperation with the OS.

As depicted, the auto-batcher 124 includes four modules: a GUI-basedtask tracker 126, a batchable task determiner 128, a batchable taskconfirmer 130, and a batch performer 132. Each of these modules isdescribed generally below, but each are discussed in more details in thefollowing section about the auto-batcher 124.

In an on-going basis, the GUI-based task tracker 126 monitors,pre-processes, and records the tasks or actions that the user 106performs within the GUI-based environment of the GUI-based OS 116. Thisincludes tasks performed with one or more GUI-based applications, suchas applications 120 and 122. More particularly, the GUI-based tasktracker 126 tracks the user's actions across multiple GUI-basedapplications. Tracked actions may include for example, clicking on abutton, double-clicking on an icon, selecting an item in a list,entering values in an input box, right-clicking, and the like.

The GUI-based task tracker 126 pre-processes the tracked tasks toorganize and identify them. In addition, the GUI-based task tracker 126stores the tracked and/or pre-processed tasks in the memory 114 and/orthe storage system 110.

In an on-going basis, as the tasks are tracked, the batchable taskdeterminer 128 detects whether batchable tasks have been performed bythe user 106. The batchable task determiner 128 detects a pattern ofrepeated tasks in the incoming stream of tracked tasks. Once itindicates that a set of repeated tasks may be batchable, the batchabletask determiner 128 predicts a forthcoming sequence of variant portionsof future iterations of the batchable tasks.

The batchable task confirmer 130 has the user 106 verify the determinedbatchable tasks and the predictions about the forthcoming sequence ofvariable portions of such tasks. Also, the batchable task confirmer 130may allow the user to correct the predictions.

With a confirmation or correction from the user, the batch performer 132then generates the remainder of the batch and performs the generatedbatch automatically (i.e., without user interaction). Indeed, the batch,when performed, may act over multiple GUI-based applications, such asGUI-based applications 120 and 122. That is, the batch may includeactions performed on, by, in, or with multiple GUI-based applications.

Example Auto-Batcher

FIG. 2 illustrates more details of at least one embodiment of theauto-batcher 124, which was introduced and briefly described in thecontext of FIG. 1. As noted above, the auto-batcher 124 includes atleast four modules: the GUI-based task tracker 126, the batchable taskdeterminer 128, the batchable task confirmer 130, and the batchperformer 132.

The GUI-based task tracker 126 monitors, pre-processes, and records theuser's GUI-based tasks. The GUI-based task tracker 126 includes at leasttwo sub-modules: a user-operation capturer 202 and a GUI-based taskparser 204.

Generally, the GUI-based task tracker 126 operates in the background ofthe GUI-based OS 116. Because of this, the user 106 is typically unawarethat the user-operation capturer 202 is silently observing and recordingthe actions performed by the user as she goes about performing varioustasks via inputs, such as those from a keyboard and a mouse. Regardlessof which application is being used (e.g., GUI-based applications 120and/or 122), the user-operation capturer 202 may capture all GUI-basedtasks of the user. Indeed, the GUI-based task tracker 126 may track theuser's actions across multiple GUI-based applications, such as GUI-basedapplications 120 and 122. Tracked tasks may include inputs from thekeyboard, mouse, touchscreen, or other such inputs used for selectionsof options of GUI-based applications.

Rather than an application-specific function, an OS-wide hook may beused to capture the input (e.g., keyboard and/or mouse) events toimplement the described techniques for automatically batching GUI-basedtasks. Generally, OS-wide hooks, using application programminginterfaces (APIs), identify and store the detail information of a GUIelement, such as the element type, element name, and its parentcomponents information. Examples of such APIs are part of the UIautomation/accessibility framework, such as Microsoft ActiveAccessibility (MSAA) and its successor, the Microsoft UI Automation(UTA), each of which is supported by various GUI-based operatingsystems. Such frameworks are general interfaces for most GUI-basedapplications and can get the information of basic GUI-based elements,such as the button, link, menu, input box, file, window, etc.

Using GUI-element identification hooks, the user-operation capturer 202gets information about the GUI-based element (e.g., the button, link,menu, input box, file, window, etc.) that is the subject of the user'sGUI-based actions. This GUI-based element is called the operationtarget.

The user-operation capturer 202 detects, tracks, and records theoperation types (e.g., the mouse/keyboard events) and operation element(e.g., which button/menu in which window is clicked). With thisinformation, the user-operation capturer 202 stores a uniqueidentification of each user operation in a defined format. Eachoperation record, as user operation is called, is stored as acombination of the operation type and the operation element. Thestructure of the operation record enables a unique representation of theoperation target and an operation type so as to indicate the userinteraction. Implementations may use new or existing formats of theoperation records. Suitable existing formats include those offered byMSAA or UTA.

The operation-record format contains the process name, main windowinformation, and frame window information, together with a hierarchytree structure of the element. For example, the element of an “OK”button in the “Save as . . . ” dialog of a specific application called“X” may have the process name “WinX”, the main window of “X”, framewindow to be the “Save as . . . ” dialog, and a hierarchy tree structurefrom the frame window to the “OK” button. With this data structure, thetarget element may be accessed later when tasks are automaticallyperformed as part of a batch.

The GUI-based task parser 204 receives a series of operation records assuch records are being captured by the user-operation capturer 202. TheGUI-based task parser 204 analyzes this operation-record sequence toparse the invariant portions of each task from the variant portions.Herein, the invariant portion is called a logkey and the variant portionis called a parameter. The invariant portion of a task includes thatportion that remains essentially the same each time the task isrepeatedly performed. The variant portion of a task includes thatportion that changes each time the task is repeatedly performed. Forexample, if the repeated task includes re-sizing an image, the actionsinvolving re-sizing are invariant while the specific image beingre-sized varies for each repetition of the task.

Below is an expanded view of the “re-sizing” example. An exampleoperation-record sequence of the repetitive task of editing a collectionof digital photos with the same “re-size” operation is as follows:

-   -   Double click on a file; name: sea.jpg; index: 1; main window:        d:\pic; frame window: file list view; . . . .    -   Click on button; name: re-size; button tree: Home—Image—re-size;        main window: Paint; Window name: sea.jpg; process: xpaint . . .        .    -   . . .    -   Double click on a file; name: mountain.jpg; index: 2; main        window: d:\pic; frame window: file list view; . . . .    -   Click on button; name: re-size; button tree: Home—Image—re-size;        main window: Paint; Window name: mountain.jpg; process: xpaint .        . . .    -   . . .    -   Double click on a file; name: flower.jpg; index: 3; main window:        d:\pic; frame window: file list view; . . . .    -   Click on button; name: re-size; button tree: Home—Image—re-size;        main window: Paint; Window name: flower.jpg; process: xpaint . .        . .    -   . . .

Example Operation Sequence 1

In the Example Operation Sequence 1 shown above, the invariant part(i.e., logkey) of each repeated task is shown without the underline.This invariant part indicates the general operation of the task. Thevariant part (i.e., parameters) is shown with underline. The variantpart indicates the specific information among similar repeatedoperations.

In different types of batchable tasks, the location and arrangement ofthe logkey and parameters may be different. The logkey and parametersare typically distinguished by comparing and contrasting the operationsub-sequences side-by-side. A sub-sequence is a chronologically adjacentgrouping of one or more tasks. For example, in the above ExampleOperation Sequence 1, through comparing the operation sub-sequencesside-by-side, the GUI-based task parser 204 is able to determine theinvariant and variant parts among the sub-sequences and label themappropriately as logkey or parameter. The parameters are “1”, “sea.jpg,”“2”, “mountain.jpg,” “3”, and “flower.jpg.” The remainder of each taskrepresents the logkey.

From observed patterns in collected records, the GUI-based task parser204 may be pre-programmed to look at some fixed locations for potentialparameters, which can assist in separating the logkey and parameters.Some of the known fixed locations for potential parameters include (byway of example only and not limitation):

-   -   The file name segment;    -   The index;    -   The value of an element;    -   The user input value;    -   The items in a list; or    -   The items in a selection menu.

By separating the logkey and parameters, the GUI-based task parser 204generates a logkey sequence. From this, the GUI-based task parser 204may map the essentially same logkeys to the same identification (e.g.,ID) label. In this way, the above Example Operation Sequence 1 is mappedinto this ID sequence:

ID1

ID2

. . .

ID1

ID2

. . .

ID1

ID2

. . .

Similarly, the GUI-based task parser 204 may map the parameters. In thisway, the above Example Operation Sequence 1 is mapped into thisparameter sequence:

Name: sea.jpq; index: 1

Window name: sea.jpq

. . .

Name: mountain.jpq; index: 2

Window name: mountain.jpq

. . .

Name: flower.jpg; index: 3

Window name: flower.jpg

. . .

Using the captured and pre-processed operation sequence from theGUI-based task tracker 126, the batchable task determiner 128 detectsbatchable tasks and attempts to anticipate the operations that the usermight perform next in the batch. The batchable task determiner 128includes at least two sub-modules: a task pattern detector 206 and aparameter projector 208.

The task pattern detector 206 uses the logkey sequence, generated hereby the GUI-based task parser 204, to detect the repetition pattern ofsub-sequences inside the operation sequence. When such a pattern isdetected, the tasks in that sub-sequence pattern are batchable. That is,the tasks of a detected pattern are capable of being batched.

The task pattern detector 206 determines that there is a pattern in thesequence when it detects at least two chronologically adjacentsub-sequences having the same set of logkey IDs. In other words, thetask pattern detector 206 declares a pattern when the operation sequenceshows at least two sets of sub-sequent tasks in succession having thesame repeated pattern of logkey IDs.

Some implementations may employ a classical repetition detectionapproach, called the Karp-Miller-Rosenberg (KMR) algorithm, to detectthe repetition pattern inside a long sequence. To illustrate further,consider this Example Operation Sequence 2: [ID1, ID2, ID3, ID4, ID5,ID2, ID3, ID4]. Using the classical repetition detection approach, thetask pattern detector 206 detects the repetition pattern of sub-sequence[ID2, ID3, ID4] of the Example Operation Sequence 2 and that pattern hasa repetition length of three. If the next operation in the ExampleOperation Sequence 2 is ID5, then the task pattern detector 206 detectsthe following as the repetition pattern: [ID2, ID3, ID4, ID5].

In some implementations, the task pattern detector 206 may use theclassical repetition detection approach with some constraints forfiltering results. Those constraints may include:

-   -   Timeliness Constraint: Since the operation sequence is being        analyzed in real or near real time, a pattern is declared when        the sub-sequence at the end of the sequence being analyzed is        part of the pattern.    -   Batchability Constraint: To be designated a pattern, the        repeated sub-sequences are adjacent to each other. That is, if        there are other operations between two otherwise identical        sub-sequences, then the two sub-sequences are not designated as        a pattern.

To illustrate the affect of the timeliness constraint, consider thisExample Operation Sequence 3: [ID1, ID2, ID3, ID1, ID2, ID3, ID1, ID2,ID3, ID6, ID5, ID2, ID4, ID7, ID6, ID2]. Based upon the timelinessconstraint, the task pattern detector 206 does not detect a pattern inthe Example Operation Sequence 3 because the end of the sequence doesnot include any pattern. The pattern of [ID1, ID2, ID3] exists in thesequence, but since it is not at the end of the sequence being analyzed,it is not designated a pattern. The imposition of this timelinessconstraint prevents the task pattern detector 206 from detecting apattern of tasks that the user is no longer performing and, thus, isprobably not interested in.

To illustrate the affect of the batchability constraint, consider thisExample Operation Sequence 4: [ID1, ID2, ID3, ID8, ID1, ID2, ID3, ID7,ID1, ID2, ID3, ID6, ID1, ID2, ID3]. Based upon the batchabilityconstraint, the task pattern detector 206 does not detect a pattern inthe Example Operation Sequence 4 because an operation (e.g., ID8, ID7,or ID6) intercedes the repeated pattern of [ID1, ID2, ID3]. Theimposition of this batchability constraint helps ensure that the taskpattern detector 206 detects patterns that are worthwhile to batch. Theintercession of non-repeating tasks (e.g., ID8, ID7, or ID6) betweenrepeating patterns means that any resulting batch would be short and,thus, the user would save little time using such a batch.

Once a pattern is detected, the parameter projector 208 then predictsthe parameters of the future operations in the anticipated batch. Thatis, the parameter projector 208 projects what the next parameters willbe for one or more of the next operations that the user has not yetperformed. That includes predicting the specific values expected for theparameters of forthcoming iterations of the batchable tasks. In someimplementations, the parameter projector 208 may make these predictionsbased upon the category or classification of the parameters and/ortasks.

Based upon a defined set of assumptions or rules for a class ofparameters, the parameter projector 208 may project the expected valuesof the parameters. For example, when the parameter is a file name andtwo or three files of a specified folder have been the subject of thetasks thus far, a reasonable rule or assumption might predict that theremaining files in that folder may be the subject of the batch. Theserules or assumptions may also be called parameter sequences herein.

The following are examples of some parameter sequences that theparameter projector 208 may use (by way of example only and notlimitation):

-   -   Number increasing/decreasing. For example, 1, 2, 3, 4, 5 . . . ;        or 100, 99, 98, 97 . . . .    -   Number increasing/decreasing with a fixed value. For example, 1,        3, 5, 7, 9 . . . .    -   Characters increasing/decreasing. For example, a, b, c, d . . .        ; or A, B, C, D . . . .    -   Month, week date increasing/decreasing. For example, Monday,        Tuesday, Wednesday . . . .    -   Value mappings inside the parameters. For example, the file        saving name can be the same or similar to the file opened name.    -   Value mappings inside the parameters with predictable changes.        For example, the file-saving name may have a fixed-change from        the file opening name.    -   Any combination of the above rules.

The parameter projector 208 may be pre-loaded with a set of useful knownparameter sequences. In addition, the user 106 may be offered anopportunity to customize the existing parameter sequences and/orgenerate new parameter sequences to match her needs. The predefined,user-customized, and/or user-generated parameter sequences are stored inthe memory 114 and/or storage system 110 of the computer device 102.

Once a pattern is detected and the parameters are projected, thebatchable task confirmer 130 confirms the predicted batchable tasks withthe user 106 or allows the user to correct the predictions. Thebatchable task confirmer 130 includes at least two sub-modules: aprojection verifier 210 and a projection corrector 212.

The projection verifier 210 notifies the user that an “auto-batch” isavailable for verification. This notification may be accomplished viaany one of several suitable mechanisms. For example, an icon may appearin a status tray, a specific audible notification may sound, and/or adialog box may appear on screen.

In response to the notification, the user may choose to engage in theverification process. As part of that process, the projection verifier210 visually steps the user through the projected next tasks in thebatch. That is, the projection verifier 210 shows the user, on screen,what the next action is that the user would have performed as part ofthe detected repeated pattern of tasks.

To illustrate how the projection verifier 210 may perform itsverification, presume that the repetition pattern is [ID1, ID2, . . . ,IDn] and the predicted parameters are [P1, P2, . . . , Pn]. From this,the predicated combination is represented by this operation sequence:[(ID1, P1), (ID2, P2), . . . , (IDn, Pn)]. With that predicted operationsequence, the projection verifier 210 locates the next operation target(e.g., a drop-down list in a particular application) and highlights theoperation elements (i.e., a particular element) of the predicted futureoperations. For example, if the next operation target is an input box,the projection verifier 210 locates that input box and shows thepredicted value in the box.

The projection verifier 210 may get verification from the user via aconventional user-input dialog box. Alternatively, the projectionverifier 210 may get verification from the user by how the userinteracts during the visual presentation of the projected next targetoperation. It may be presumed that the user agrees or disagrees with theanticipated batch task by how the user follows or fails to follow avisual simulation of the anticipated batch task. Here, “following” mayinclude the user providing coordinating user input, such as theuser-controlled cursor moving near or along with the simulated cursorcontrolled by the projection verifier 210.

With one or more implementations, the batch verifier may draw a coloredshape (e.g., red rectangle) around the located GUI element that is thetarget of the projected target operation. This colored shape highlightsthe target operation and acts as a guide to the user. If the user doesnot follow the highlighted target to do the operations for a definednumber of steps (e.g., three steps), then the projection verifier 210may assume that the prediction is wrong and guidance ends.

If the prediction is wrong, the projection corrector 212 may ask theuser to correct the projection. The user may do this, for example, bymodeling the missing or incorrect GUI-based tasks or by correcting theparameter sequence used by the parameter projector 208. Once corrected,the parameter projector 208 may generate a new projection based upon thecorrection.

If the user follows the highlight target and the predicted values to dothe operations for one task or a major part of one task, then theprojection verifier 210 concludes that the prediction is correct. Inthis case, the projection verifier 210 may ask the user to confirmwhether to automatically complete the succeeding tasks (e.g., the batch)on her behalf.

Instead of verifying or correcting, if the user ignores the notificationor does not confirm the projections, then the present projections arediscarded. However, the user may be notified again once a new projectionis ready based upon the existing or a newly detected pattern ofGUI-based tasks.

With the user's confirmation or correction, the batch performer 132performs the batch based upon the verified or corrected projections (ormore simply, the “confirmed projections”). The batch performer 132includes at least four sub-modules: a parameter range determiner 214, anoperation target locator 216, a batch generator 218, and a user-inputemulator 220.

The parameter range determiner 214 detects and confirms the range of theparameters. Based upon context, the parameter range determiner 214 maydetect the range or upper/lower limits of the parameters. For example,by noticing that there are fifty digital photos in a target subfolder,the parameter range determiner 214 may determine that these fiftydigital photos form the range or limit for the parameter. Then, theparameter range determiner 214 may ask the user to confirm this detectedrange. The user also has a chance here to correct the range if thedetected range is not what the user desires. Alternatively, the user mayjust be asked for the range without any detection being performed.

To mimic the user actually performing the remaining part of the batchtasks, the operation target locator 216 locates the target elements ofthe operations of the generated batch. That is, the location of thetarget element (e.g., buttons, input boxes, etc.) is determined within arelative context. As opposed to an absolute context (e.g., coordinatesof pixels on a screen or in a window), the relative context is where aparticular element is located regardless of its absolute location. Forexample, a particular button may be located regardless of where it is ona screen at the time.

To illustrate, consider this repetition pattern [ID1, ID2, . . . , IDn]and this predicted parameter set [P1, P2, . . . , Pn]. When the two arecombined, this predicted set of operations is generated: [(ID1, P1),(ID2, P2), . . . , (IDn, Pn)]. The operation target locator 216 mapseach of these IDs back to a logkey (i.e., the invariable part of aGUI-based task). The operation target locator 216 generates an operationrecord by adding the parameters (Pn) to the IDs. Herein, this is calledthe predicted operation record. The predicted operation records may beformatted in accordance with the operation-record format discussedabove. Each predicted record uniquely represents its operation target,together with an operation type to indicate the user interaction.

Within the operation-record format, each predicted operation recordincludes the process name, main window information, frame windowinformation, and a hierarchy tree structure of the element. With thisdata structure, the target element on the system can be found. Theoperation type is used to indicate the user interaction, such as aright-click, left-click, double-click, drag, mouse move, keyboard input,etc.

With the predicted operation record, the batch generator 218 generates apredicted operation sequence, which may be called the predicted batch.The predicted batch may be stored in the memory 114 and/or storagesystem 110 of the computer device 102.

When each GUI-based task of a predicted operation record is preformed,the user-input emulator 220 mimics the appearance of a user controllingthe input. This user-input emulation is more pleasing to the user andprovides the user with feedback regarding what actions are taking placeautomatically. Exactly what occurs depends upon the operation type ofthe operation target of the predicted operation record beingautomatically performed for the user.

Consider these examples:

-   -   If the operation being performed is a click operation (e.g.,        left-click, right click, or double-click), the user-input        emulator 220 determines the distance between the current mouse        position and the target element. The user-input emulator 220        then inserts proper mouse moves and directs the mouse to move        from its current position to the target element. At that point,        the user-input emulator 220 generates a mouse click event on the        element.    -   If the operation being performed is a move operation, the        user-input emulator 220 generates the move events for smooth        control.    -   If the operation being performed is a drag, the user-input        emulator 220 moves the mouse to the target element and generates        a left mouse down event. At that point, the user-input emulator        220 moves to the destination and generates a left mouse up        event.    -   If the operation being performed is a keyboard operation, the        user-input emulator 220 will directly send the input value to        the target input area.

The user-input emulator 220 may implement the click/double-click/moveoperations by generating system-level mouse events. For example, a clickconsists of one mouse down and one mouse up with a proper time interval,and a double-click consists of a first mouse down, a first mouse up, asecond mouse down, and a second mouse up.

Example Processes

FIGS. 3 and 4 are flow diagrams illustrating example processes 300 and400 that implement the techniques described herein for auto-batchingGUI-based tasks. These processes are each illustrated as a collection ofblocks in a logical flow graph, which represents a sequence ofoperations that can be implemented in hardware, software, firmware, or acombination thereof. In the context of software, the blocks representcomputer instructions stored on one or more computer-readable storagemedia that, when executed by one or more processors of such a computer,perform the recited operations. Note that the order in which theprocesses are described is not intended to be construed as a limitation,and any number of the described process blocks can be combined in anyorder to implement the processes or an alternate process. Additionally,individual blocks may be deleted from the processes without departingfrom the spirit and scope of the subject matter described herein.

FIG. 3 illustrates the example process 300 for auto-batching GUI-basedtasks. The process 300 is performed, at least in part, by a computingdevice or system, which includes, for example, the computing system 102of FIG. 1. The computing device or system is configured to facilitateauto-batching of GUI-based tasks within the context of a GUI-basedenvironment. The computing device or system so configured qualifies as aparticular machine or apparatus.

As shown here, the process 300 begins with operation 302, where thecomputing device tracks the GUI-based tasks or actions performed by theuser as part of a GUI-based OS environment. This operation is thedefault or baseline operation performed by the process. The trackingincludes monitoring, pre-processing, and recording the user's GUI-basedtasks.

As part of operation 302, the computing device tracks the GUI-basedtasks of the user within the context of the GUI-based environment. Thecomputing device detects and captures user input as discrete tasks oractions performed by the user via a graphical user interface. This maybe accomplished, for example, by having a hook that is running incooperation with the OS for identifying the GUI-elements of the OS orGUI-based application.

In addition, the computing device converts the captured tasks into apre-defined format, such as the operation-record format alreadydiscussed. Then, the components of the formatted captured tasks areparsed, and portions are labeled as being invariant or variant relativeto other tasks performed by the user. That is, the computing deviceparses and labels each of the tracked GUI-based tasks into an invariantportion and one or more variant portions, wherein the invariant portionof like tasks do not differ between like tasks and the variant portionsof like tasks differ between like tasks. Herein, the invariant portionsof a task are called logkeys and the variant portions are calledparameters.

At operation 304, the computing device detects batchable tasks. Thecomputing device examines the invariant portion of each task of asequence of GUI-based tasks and detects, in the sequence, a repeatedsub-sequence of tasks having a common pattern of invariant portions.

By comparing the invariant portions of tasks, the computing devicedetermines whether the user is manually performing a repeated pattern oflike tasks. As indicated by the dashed arrow, operation 304 returns backto the baseline tracking operation 302 until a repeated pattern isdetected. Conversely, once a repeated pattern is detected, this processproceeds to the next operation, which is 306.

At operation 306, the computing device anticipates one or more GUI-basedtasks not-yet-performed by the user. To do this, the computing devicesets the values of the parameters and thereby effectively predicts theparameters of the future operations in the anticipated batch. Based, atleast in part, upon known pre-defined, user-customized, and/oruser-defined sequences of parameters, the computing device projects whatthe next parameters will be for the next operations that the user hasyet to perform. As indicated by the dashed arrow, this operation returnsback to the baseline tracking operation 302, if no appropriate parametersequence is found in order to project the parameters of the detectedpattern. Conversely, once the appropriate parameters are projected, thisprocess proceeds to the next operation, which is 308.

At operation 308, the computing device verifies with the user that theone or more anticipated GUI-based tasks match forthcoming GUI-basedtasks that the user intends to perform. To do this, the computing deviceverifies that the projected parameters accurately predict the actionsthe user was intending to perform next. This may be accomplished bydemonstrating the next actions for the user and asking the user toconfirm. Alternatively, this may be accomplished by asking the user tofollow the demonstrated actions with user input.

At operation 310, the computing device receives an indication from theuser, during the verification process, as to whether the parameters ofthe demonstrated actions are accurate. If verified, the process 300proceeds to operation 314. If not verified, the process 300 may returnto operation 302 to track the user's tasks again. A failure of the userto respond within a defined period (e.g., 5 seconds) may be presumed tobe non-verification. Alternatively, the user may be offered anopportunity to correct a non-verified projection. In this instance, theprocess 300 may proceed to operation 312 for the user to correct thebatch.

At operation 312, the user corrects or adjusts the parameter sequenceused by operation 306 to match the pattern that the user intends to use.By utilizing the user-provided or user-corrected parameter sequence,this operation may perform the projections of operation 306. From here,the process 300 may proceed to operation 314.

At operation 314, the computing device performs a batch of the verifiedGUI-based tasks, visually simulating a user providing input inperforming the GUI-based tasks of the batch. In other words, thecomputing device performs the batch based upon the verified or correctedprojections. FIG. 4 provides more details about how that is done. Oncethe batch is performed, the process 300 returns back to baselineoperation 302 to continue to track the user's GUI-based tasks.

FIG. 4 illustrates the example process 400 for auto-batching GUI-basedtasks. The process 400 is performed, at least in part, by a computingdevice or system, which includes, for example, the computing system 102of FIG. 1. The computing device or system is configured to facilitatethe auto-batching of GUI-based tasks within the context of a GUI-basedenvironment. The computing device or system so configured qualifies as aparticular machine or apparatus.

As shown here, the process 400 begins with operation 402, where thecomputing device detects and confirms a range of parameters. Based uponcontext, the computing device may detect a range, or upper/lower limits,of the parameters. After detecting the upper and/or lower limit of theparameters, the computing device may ask the user to confirm thisdetected limit.

At operation 404, the computing device locates the target elements ofthe operations of the generated batch in order to be able to later mimicthe user performing the batch tasks. The computing device maps the IDsof the detected repetition pattern to their logkeys. The computingdevice then combines the predicted parameters with the mapped logkeys toproduce predicted operation records. The predicted operation records maybe formatted in accordance with the operation-record format alreadydiscussed.

Using the predicted operation records, at operation 406, the computingdevice generates a predicted operation sequence, which may be called thepredicted batch. The generated batch may be stored in the memory 114 orstorage system 110 of the computer device 102.

At operation 408, for each GUI-based task of a predicted operationrecord, the user-input emulator 220 determines how to emulate a usercontrolling the input. In this way, while the batch is beingautomatically performed, it appears that a user is controlling theinput. For example, it appears that the user is moving the cursoron-screen and clicking buttons.

At operation 410, the computing device performs the predicted batch anddoes so without user interaction or intervention. That is, it does soautomatically.

CONCLUDING NOTES

As used in this application, the terms “component,” “module,” “system,”“interface,” or the like, are generally intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of example, both an application running on acontroller and the controller can be a component. One or more componentsmay reside within a process and/or thread of execution and a componentmay be localized on one computer and/or distributed between two or morecomputers.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture, using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof, to control a computer to implement the disclosedsubject matter. The term “article of manufacture,” as used herein, isintended to encompass a computer program accessible from anycomputer-readable device, carrier, or media.

Computer-readable media includes, at least, two types ofcomputer-readable media, namely computer storage media andcommunications media.

Computer storage media includes volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer readable instructions, data structures,program modules, or other data. Computer storage media includes, but isnot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other non-transmission mediumthat can be used to store information for access by a computing device.

In contrast, communication media may embody computer readableinstructions, data structures, program modules, or other data in amodulated data signal, such as a carrier wave, or other transmissionmechanism. As defined herein, computer storage media does not includecommunication media.

As used in this application, the term “or” is intended to mean aninclusive “or” rather than an exclusive “or”. For example, unlessspecified otherwise or clear from context, “X employs A or B” isintended to mean any of the natural inclusive permutations. That is, ifX employs A; X employs B; or X employs both A and B, then “X employs Aor B” is satisfied under any of the foregoing instances. In addition,the articles “a” and “an”, as used in this application and the appendedclaims, should generally be construed to mean “one or more”, unlessspecified otherwise or clear from context to be directed to a singularform.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as example forms ofimplementing the claims.

1. (canceled)
 2. A method comprising: producing a batch of GUI-based tasks; and performing the batch of the GUI-based tasks.
 3. The method as recited in claim 2, the batch of GUI-based tasks including GUI-based tasks from at least two GUI-based applications.
 4. The method as recited in claim 3 in a GUI-based operating system environment.
 5. The method as recited in claim 2, further comprising prompting for verification to include additional tasks in the batch of the GUI-based tasks.
 6. The method as recited in claim 2, further comprising determining a parameter value for a particular GUI-based task within the batch of the GUI-based tasks.
 7. The method as recited in claim 6, wherein the determining the parameter value is based at least in part on input from a user.
 8. The method as recited in claim 6, wherein the determining the parameter value is based at least in part on parameters of previously recorded GUI-based tasks.
 9. The method as recited in claim 2, further comprising recording GUI-based tasks performed within a context of a GUI-based environment.
 10. The method as recited in claim 2, further comprising locating a target element of a particular GUI-based task within the batch of the GUI-based tasks.
 11. A method comprising: generating a sequence of GUI-based tasks for future execution; and executing the sequence of GUI-based tasks.
 12. The method as recited in claim 11, wherein the GUI-based tasks in the sequence of GUI-based tasks include a first GUI-based task from a first application and a second GUI-based task from a second application.
 13. The method as recited in claim 11, further comprising prompting for verification that the generated sequence of GUI-based tasks for future execution represents additional tasks that a user intends be performed.
 14. The method as recited in claim 11, further comprising determining a parameter value for a particular GUI-based task within the sequence of the GUI-based tasks.
 15. The method as recited in claim 11, further comprising recording GUI-based tasks.
 16. The method as recited in claim 11, further comprising configuring a GUI-based environment for execution of the GUI-based tasks.
 17. The method as recited in claim 11, further comprising locating a target element of a particular GUI-based task within the sequence of the GUI-based tasks for future execution.
 18. A system comprising: a processor; a batch module configured to be executed at least in part by the processor to: produce a batch of GUI-based tasks; and perform the batch of the GUI-based tasks.
 19. The system as recited in claim 18, wherein the batch module is configured for automatically batching GUI-based tasks.
 20. The system as recited in claim 18, wherein the batch module is configured to record GUI-based tasks.
 21. The system as recited in claim 18, wherein the GUI-based tasks include a first GUI-based task from a first application and a second GUI-based task from a second application. 